Method and System for Providing Position or Movement Information for Controlling At Least One Function of an Environment

ABSTRACT

A method for providing position information for controlling at least one function of an environment provided with an environment coordinate system comprises the steps of receiving image information associated with at least one image of at least one part of the environment captured by a camera of a mobile device, the mobile device being separate from the environment, determining a position associated with the mobile device in the environment coordinate system according to the image information associated with the at least one image, and providing position information indicative of the position associated with the mobile device in the environment coordinate system to control at least one function of the environment according to the position. There is also provided a method and system which provides movement information indicative of a movement associated with the mobile device relative to the environment coordinate system to control at least one function of the environment according to the movement.

BACKGROUND

Self-calibration of camera positions relative to a ground plane for acamera attached to an environment is known. Obtaining information aboutan occupying item in an environment is also known. For example, imagesof an area in the environment in which the occupying item may besituated and employs a trained neural network to classify the item inthe images. However, what is needed is to mount the display device to anenvironment and to control a function of the environment based thereon.

SUMMARY

It is an object of the invention to provide a method and system whichenables a user to provide information which is appropriate forcontrolling at least one function of an environment based on a positionor movement.

According to a first aspect, there is provided a method for providingposition information for controlling at least one function of anenvironment provided with an environment coordinate system, the methodcomprising receiving image information associated with at least oneimage of at least one part of the environment captured by a camera of amobile device, the mobile device being separate from the environment,determining a position associated with the mobile device in theenvironment coordinate system according to the image informationassociated with the at least one image, and providing positioninformation indicative of the position associated with the mobile devicein the environment coordinate system to control at least one function ofthe environment according to the position.

The invention can advantageously make use of the fact that modernenvironments often comprise, e.g., a display device mounted inside theenvironment at a known position relative to the environment and towarddrivers and/or passengers sitting in the environment, and that apersonal mobile device, such as a mobile phone or tablet computer, oftencomprises a camera.

According to the present invention, at least one function of anenvironment may be controlled based on a determined position associatedwith a mobile device comprising a camera, which position is determinedby using the camera to capture at least one image of a part of theenvironment. For example, the captured part of the environment includesa visual content displayed on a display device mounted to theenvironment. In this way, the invention provides a method and systemwhich enables a user to determine a position inside an environment forcontrolling at least one function of the environment based on thedetermined position. For example, a function of a sound system, alighting system and/or an air conditioning of the environment may becontrolled according to the position of the mobile device.

To this end, according to an embodiment, the invention proposes todetermine a position of a mobile device in a coordinate system of anenvironment by using, for example, a display device mounted to theenvironment, a camera of the mobile device, and images of visualcontents displayed on the display device captured by the camera of themobile device. Furthermore, at least one function of the environment iscontrolled based on the determined position. For example, the functionis related to a sub-system of the environment, such as a sound system,an ambient light system, an air conditioning system, or theircombination.

According to an embodiment, the method further comprises providing orreceiving a position of the at least one part of the environment in theenvironment coordinate system.

According to an embodiment, the at least one part of the environmentcomprises a visual content displayed on a display device mounted to theenvironment.

According to an embodiment, determining the position associated with themobile device in the environment coordinate system comprises determininga camera position of the camera relative to the at least one part of theenvironment when capturing the at least one image according to the imageinformation associated with the at least one image, and determining theposition associated with the mobile device in the environment coordinatesystem according to the camera position and the position of the at leastone part of the environment in the environment coordinate system.

According to a further embodiment, the at least one image is a firstimage and the camera is a first camera, and determining the positionassociated with the mobile device in the environment coordinate systemcomprises receiving image information associated with a second image ofthe at least one part of the environment captured by a second camera ofthe mobile device, and reconstructing a model of the at least one partof the environment according to the image information associated withthe first and second images.

In a further development, the method further comprises receiving imageinformation associated with a third image captured by a third camera ofthe mobile device, wherein the third image captures at least a part ofthe environment represented by at least part of the reconstructed model(this at least a part of the environment could be different from the atleast one part of the environment mentioned above; however, the at leasta part of the environment is included in the reconstructed model),determining a camera position of the third camera relative to thereconstructed model when capturing the third image according to the atleast part of the reconstructed model and image information associatedwith the third image, and determining the position associated with themobile device in the environment coordinate system according to thecamera position of the third camera.

According to an embodiment, the method further comprises determining acamera position of the first camera relative to the at least one part ofthe environment when capturing the first image according to at leastpart of the image information associated with the first image, anddetermining a camera position of the second camera relative to the atleast one part of the environment when capturing the second imageaccording to at least part of the image information associated with thesecond image.

According to a further development, the at least one part of theenvironment is a first part of the environment, and the method furthercomprises receiving image information associated with a fourth imagecaptured by a fourth camera of the mobile device, wherein the fourthimage captures at least part of the first part of the environment and atleast a second part of the environment, and extending the reconstructedmodel to include the first part of the environment and the second partof the environment.

Preferably, the reconstructed model at least describes depth informationof at least part of the environment.

According to an embodiment, at least two of the first camera, the secondcamera, the third camera, and the fourth camera are the same camera.

According to another embodiment, at least two of the first camera, thesecond camera, the third camera, and the fourth camera are differentcameras.

According to an embodiment, the at least one function is a functionwhich is related to at least one of a sound system of the environment,an ambient light system of the environment, and an air conditioningsystem of the environment, and which is controlled according to theposition.

According to the first aspect, there is also provided a system forproviding position information for controlling at least one function ofan environment provided with an environment coordinate system, thesystem comprising a first processing device configured to receive imageinformation associated with at least one image of at least one part ofthe environment captured by a camera of a mobile device, the mobiledevice being separate from the environment, the first processing deviceconfigured to determine a position associated with the mobile device inthe environment coordinate system according to the image informationassociated with the at least one image, and a second processing deviceconfigured to provide position information indicative of the positionassociated with the mobile device in the environment coordinate systemto control at least one function of the environment according to theposition.

According to a second aspect, there is provided a system for providingmovement information for controlling at least one function of anenvironment provided with an environment coordinate system, comprising afirst processing device configured to receive image informationassociated with at least two images captured by at least one camera of amobile device, the mobile device being separate from the environment,and each of the at least two images capturing at least one part of theenvironment, the first processing device configured to determine amovement associated with the mobile device relative to the environmentcoordinate system according to the image information associated with theat least two images, and a second processing device configured toprovide movement information indicative of the movement associated withthe mobile device relative to the environment coordinate system tocontrol at least one function of the environment according to themovement.

The following embodiments as well as other embodiments described hereinare equivalently applicable in connection with, both, the first andsecond aspect.

According to an embodiment, the first processing device and the secondprocessing device are the same processing device.

According to an embodiment, at least one of the first and secondprocessing devices is comprised in the mobile device. According toanother embodiment, at least one of the first and second processingdevices is comprised in the environment. According to a furtherembodiment, at least one of the first and second processing devices iscomprised in a computer device which communicates with the mobile deviceand the environment.

According to the second aspect, there is also provided a method forproviding movement information for controlling at least one function ofan environment provided with an environment coordinate system, themethod comprising receiving image information associated with at leasttwo images captured by at least one camera of a mobile device, themobile device being separate from the environment, each of the at leasttwo images capturing at least one part of the environment, determining amovement associated with the mobile device relative to the environmentcoordinate system according to the image information associated with theat least two images, and providing movement information indicative ofthe movement associated with the mobile device relative to theenvironment coordinate system to control at least one function of theenvironment according to the movement.

According to an embodiment, the movement associated with the mobiledevice relative to the environment coordinate system can be determinedaccording to two positions associated with the mobile device in theenvironment coordinate system. The two positions associated with themobile device can be determined according to the image informationassociated with the at least two images respectively. The at least onepart of the environment captured in the each of the at least two imagesmay have common parts or be different.

According to a further embodiment, the movement associated with themobile device relative to the environment coordinate system can bedetermined according to two camera positions of the at least one camerawhile capturing the at least two images.

Aspects and embodiments of the invention described above or in thefollowing with respect to the first aspect related to determiningpositions associated with the mobile device can equivalently also beapplied to the second aspect related to determining a movementassociated with the mobile device.

According to an embodiment, the movement associated with the mobiledevice relative to the environment coordinate system can be determinedaccording to image positions of the at least one part of the environmentcaptured in the each of the at least two images. This requires that theat least one part of the environment captured in the each of the atleast two images are same or have common parts. However, this does notrequire known positions of the at least one part of the environment inthe environment coordinate system.

According to the present invention, at least one function of anenvironment may be controlled based on a determined movement associatedwith a mobile device. For example, a function of a sound system, alighting system and/or an air conditioning of the environment may becontrolled according to the movement of the mobile device. The movementmay be used to change volume of the sound system. The lighting systemmay change a current illuminated position to a new illuminated positionaccording to the determined movement of the mobile device. For example,the determined movement may define how far away from the currentilluminated position to the new illuminated position. In anotherexample, the determined movement may define a rotation such that thelighting system may rotate from a current illuminating direction to anew illuminating direction according to the rotation.

Preferably, the mobile device is a hand held device, such as a mobilephone, a tablet computer or a mobile computer.

For example, the at least one part of the environment has a position inthe environment coordinate system which could be provided by themanufacturer of the environment.

The at least one part of the environment captured in an image could beany physical part of the environment. The at least one part of theenvironment may include a display device mounted to the environment or apart of the display device. The at least one part of the environment mayfurther include one or more visual contents displayed on the displaydevice.

The display device is a device for visually presenting of information.The display device could be based on any displaying technologies ormaterials, such as Cathode ray tube (CRT), Light-emitting diode display(LED) and Liquid crystal display (LCD). The display device may include a2-dimensional planar display or a display having a curved shape. Thedisplay device may also be a foldable display device comprising multipleplanar sub-displays, each of which could be moved with others.

The display device may be mounted to the environment. It is preferred tobe mounted towards eyes of drivers and/or passengers sitting in theenvironment, such that the drivers and/or passengers could see visualcontents displayed on the display device. For example, the displaydevice may be mounted on a front control panel of the environment, ormounted on a back of a front seat of the environment.

The display device could be used to display menus, maps, graphical userinterfaces of software programs, etc.

The visual content is any visually perceivable information to anatomicaleyes or optical imaging devices. For example, the visual content mayemit or reflect visible light that could be captured by human eyes orcameras. The visual content may also emit or reflect invisible lightthat could be not captured by human eyes, but could be captured by acamera. The visual content could be a text, a figure, an image generatedby computers or captured by cameras, a symbol, a drawing, theircombinations, or a part of each of them. For example, the visual contentmay be a menu, a button, an icon, a digital map, a graphical userinterface of a software program, their combinations, or a part of eachof them. The visual content can be displayed on the display device.

The mobile device is portable and comprises one or more cameras. The oneor more cameras may have known positions relative to the mobile device.The mobile device could only be a camera. For example, the mobile deviceis any of the camera, the first camera, the second camera, the thirdcamera, and/or the fourth camera, i.e. the mobile device and therespective camera are the same device.

In one embodiment, the mobile device may have at least one processingdevice, such as a computing processor. In another embodiment, the mobiledevice may not have a computing processor, but may have a transmitter totransmit data (e.g. image data of images captured by the camera) toanother device (e.g. the environment). The mobile device could be, butis not limited to, a mobile phone, a tablet computer, or a laptop. Thecamera may be separate from the mobile device, but communicate with themobile device via a cable or wirelessly.

The one or more cameras are optical imaging devices that could captureimagery information of optical information.

The proposed invention can be applied with any camera providing images.It is not restricted to cameras providing color images in the RGBformat. It can also be applied to any other color format and also tomonochrome images, for example, to cameras providing images in grayscaleformat. The camera may further provide an image with depth data. Thedepth data does not need to be provided in the same resolution as the(color/grayscale) image. A camera providing an image with depth data isoften called RGB-D camera. A RGB-D camera system could be a time offlight (TOF) camera system. The camera may also capture light that isinvisible to human eye, such as infrared light. For example, the cameramay be a thermal imaging camera.

A position of a camera in the environment coordinate system may includetranslations, or rotations, or their combination, in the environmentcoordinate system.

A visual appearance of the displayed visual content describes shape,texture, geometry or their combinations. The visual appearance may ormay not include color information.

A physical geometry of an object describes size, shape, dimension,planarity, or their combinations of the object as it is in the realworld.

A resolution of the display device is the number of distinct pixels ineach dimension that can be displayed on its display area. The displaydevice may have a known resolution. The display device may further havea known physical geometry for its display area. The physical geometry ofthe display device refers to the physical geometry of the display areaof the display device. Having the pixel position of the displayed visualcontent in a coordinate system of the display device, and the resolutionand the physical geometry of the display device, a spatial relationshipbetween the displayed visual content and the display device can bedetermined. This defines a position of the displayed visual contentrelative to the display device. Further, a physical geometry of thedisplayed visual content can also be determined. The position of thedisplayed visual content in the environment coordinate system may bedetermined according to the position of the displayed visual contentrelative to the display device and a position of the display devicerelative to the environment coordinate system.

A camera position of the camera relative to the at least part of theenvironment while the first image is captured by the first camera can bedetermined according to at least part of the image. Having a physicalgeometry of the at least part of the environment, various vision basedcamera position estimation methods, such as based on 2D-3D pointcorrespondences (see, e.g., Hartley, Richard, and Andrew Zisserman.Multiple view geometry in computer vision. Vol. 2. Cambridge, 2000), canbe employed to determine the camera position.

Particularly, it is preferred to have the at least part of theenvironment to include a displayed visual content. The image of thecamera thus captures the displayed visual content. The visual contentcould provide rich texture and features (e.g. contrasting corners oredges), which will make vision based camera position estimations robust.In this example, the camera position of the camera can be determinedrelative to the display device mounted to the environment. A position ofthe display device relative to the environment coordinate system couldbe provided from the environment manufacturer, for example.

Having the known visual appearance of the displayed visual content andits physical geometry, a camera position of the camera relative to thevisual content can be determined with a correct scale factor accordingto at least part of the image information of the first image. Cameraposition estimation could be based on correspondences between imagefeatures of the captured image and corresponding features of the visualcontent displayed on the display device. The camera position in theenvironment coordinate system is then determined according to the cameraposition relative to the displayed visual content and the position ofthe displayed visual content in the environment coordinate system.

The camera position of the camera in the environment coordinate systemwhile the camera captures the image can be determined from the cameraposition relative to the at least one part of the environment and theposition of the at least one part of the environment in the environmentcoordinate system.

In one example, the position of the camera in the environment coordinatesystem could determine the position that can be provided to theenvironment to control the at least one function of the environment. Theposition associated with the mobile device may be the same as theposition of the camera in the environment coordinate system. Theposition associated with the mobile device may have a displacement inspace from the position of the camera in the environment coordinatesystem.

In another example, the position of the camera in the environmentcoordinate system may not be appropriate to specify the positionassociated with the mobile device. For example, the position of thecamera may not be a desired position to be provided to the environmentto control the at least one function.

According to an embodiment, the present invention proposes to furthercapture a second image of the at least one part of the environment by asecond camera of the mobile device (with the previously described cameraand image being a first camera and first image, respectively). Thesecond image and the first image are captured when the second camera andthe first camera are at different positions.

A camera position of the second camera relative to the at least one partof the environment while the second image is captured may be determinedaccording to the at least part of the second image. The disclosed methodof determining the camera position of the first camera may also be usedto determine the camera position of the second camera.

A model of at least one part of the environment may be reconstructedaccording to the first and second images and the camera positions of thefirst and second cameras.

In one embodiment of reconstructing the model, correspondences betweenimage features of the at least one part of the environment in the firstand second images are determined. Then, a triangulation method can beused to determine the model from the image feature correspondences andthe camera positions of the first camera and second camera, see forexample Hartley, Richard, and Andrew Zisserman. Multiple view geometryin computer vision. Vol. 2. Cambridge, 2000.

A model of an object, as described above, at least describes depthinformation of at least part of the object. The model further mayinclude one of the following attributes, but is not limited to, shape,symmetry, planarity, geometrical size, color, texture and density.

As the physical geometry of the at least one part of the environmentcaptured in the first and second images could be known, a correct scalefactor for the reconstructed model could be determined as taught, e.g.,in Davison, Andrew J., et al. “MonoSLAM: Real-time single camera SLAM.”Pattern Analysis and Machine Intelligence, IEEE Transactions on 29.6(2007): 1052-1067. The reconstructed model can be represented as 3Dvertices, polygonal faces and/or edges spanned by these vertices. Edgesand faces of the model may also be represented as splines or NURBSsurfaces.

The reconstructed model may be determined in the environment coordinatesystem. The reconstructed model may also be determined in an arbitrarycoordinate system, such as its own coordinate system. In this case, thereconstructed model can be related to the environment coordinate systembased on the position of the at least one part of the environment in theenvironment coordinate system.

According to an embodiment, a third camera of the mobile device maycapture a third image. The third image includes a part of theenvironment, which is represented by at least part of the reconstructedmodel. The part of the environment does not need to include the at leastone part of the environment captured by the first camera, or includesonly a part thereof. Particularly, the part of the environment capturedin the third image does not need to include the display device of theenvironment or any visual content displayed on the display device.

A camera position of the third camera relative to the model while thethird image is captured may be determined according to at least part ofthe reconstructed model and at least part of the third image. Forexample, a SLAM method (see Davison, Andrew J., et al. “MonoSLAM:Real-time single camera SLAM.” Pattern Analysis and MachineIntelligence, IEEE Transactions on 29.6 (2007): 1052-1067) could beemployed for determining the camera position of the third camera.

In one example, the position of the third camera in the environmentcoordinate system while the third camera captures the third image may beused to determine the position associated with the mobile device in theenvironment coordinate system. The position associated with the mobiledevice may be the same as the position of the third camera in theenvironment coordinate system. The position associated with the mobiledevice in the environment coordinate system may have a displacement inspace from the position of the third camera in the environmentcoordinate system.

The camera position of the third camera in the environment coordinatesystem can be determined from the camera position of the third camerarelative to the model and a spatial relationship between the model andthe environment coordinate system.

Furthermore, the model can further be used to initialize a vision basedSimultaneous Localization and Mapping (SLAM) method, such as describedin references [4,5,6], for tracking the camera(s) of the mobile devicein the environment coordinate system and/or extending the reconstructedmodel by adding other reconstructed parts of the environment.

For example, a fourth image of at least one second part of theenvironment is captured by a fourth camera of the mobile device. Thefourth image may further capture a part of the environment that isrepresented by at least part of the reconstructed model. Then, thereconstructed model could be extended to have the first part and secondpart of the environment based on the SLAM method. This may require tocompute a camera pose of the fourth camera relative to the model whilecapturing the fourth image based on matching image positions of the partof the environment and the at least part of the reconstructed model ofthe part of the environment.

The position associated with the mobile device can be determinedaccording to a camera position of a camera of the mobile device in theenvironment coordinate system. The camera position could be estimatedaccording to at least part of the reconstructed model and at least oneimage captured by the camera based on the SLAM method. For this, the atleast one image captures a part of the environment represented by the atleast part of the reconstructed model. The position associated with themobile device may be the same as the position of the camera while thecamera captures the at least one image.

The determined position or the determined movement associated with themobile device in the environment coordinate system can be provided tothe environment to control the at least one function of the environmentaccording to the position or the movement. The at least one function maybe related to one or more sub-systems of the environment. Sub-systems ofthe environment could be, but are not limited to, a sound system, anambient light system, and an air conditioning system.

The at least one function could control or configure one or moresub-systems of the environment according to the determined positionassociated with the mobile device in the environment coordinate system.For example, the position could specify a center position of an area inspace where the ambient light system should illuminate or the airconditioning system should monitor temperature. A listening area couldalso be specified by the position such that the sound system would becontrolled or adjusted for orientating it towards the specifiedlistening area.

The mobile device may communicate with the environment directly or via acomputer network. The mobile device may communicate with the environmentindirectly via other devices, e.g. via a web service computer.

The computer network may be a telecommunications network that connectsprocessing devices (e.g. computers) to allow communication and dataexchange between systems, software applications, and users. Theprocessing devices may be connected via cables, or wirelessly, or bothvia cables and wirelessly. For example, the computer network could be anInternet, intranet, local area network, or wide area network.

In one embodiment, the mobile device may comprise one camera. The firstcamera, the second camera, the third camera, and the fourth camera arethe same camera.

In another embodiment, the mobile device may comprise at least twocameras. At least two of the first camera, the second camera, the thirdcamera, and the fourth camera are different cameras.

In another embodiment, at least two of the first camera, the secondcamera, the third camera, and the fourth camera are the same camera.

Camera intrinsic parameters for the first camera, the second camera, thethird camera, and/or the fourth camera may be provided or calibratedbased on a camera calibration method. The camera intrinsic parametersmay be used to determine camera positions or camera movements based oncamera images.

In one embodiment, the system according to the invention could beelectronically coupled with or be a part of the mobile device. It couldbe separate from the environment. The system may communicate with theenvironment via cable, wirelessly or via a computer network. The systemcould determine the position or the movement associated with the mobiledevice in the environment coordinate system and send the position or themovement to the environment for controlling the at least one function.

In a further embodiment, the system could receive environmentinformation from the environment. The environment information mayinclude at least one of the environment coordinate system, a visualappearance of a displayed visual content, a resolution of a displaydevice used for the display, a position of the display device (i.e. theat least one part of the environment) in the environment coordinatesystem, a physical geometry of the display device (i.e. the at least onepart of the environment), or their combinations.

In another embodiment, the system could be electronically coupled withor be a part of the environment and separate from the mobile device. Thesystem may communicate with the mobile device via cable, wirelessly orvia a computer network. The mobile device could send camera images tothe system. The mobile device may further need to send camera intrinsicparameters (e.g. focal length and principal point) to the system. Thecamera intrinsic parameters may be used to determine camera positionsbased on camera images.

In another embodiment, the system could be separate from the mobiledevice and separate from the environment. The system may communicatewith the mobile device and/or the environment via cable, wirelessly orvia a computer network. The system could receive environment informationfrom the environment and receive camera images and camera intrinsicparameters from the mobile device. Then, the system could determine theposition or the movement associated with the mobile device in theenvironment coordinate system and send the position or movement to theenvironment or to one or more sub-systems of the environment related tothe at least one function for controlling the at least one function.

According to another aspect, the invention is also related to a computerprogram product comprising software code sections which are adapted toperform a method according to the invention. Particularly, the computerprogram product is contained on a computer readable medium and isnon-transitory. The software code sections may be loaded into a memoryof one or more of the processing devices as described herein.

According to an embodiment, the method may be implemented as anapplication which runs on one or more processing devices of a mobiledevice, such as a mobile phone, and which communicates directly orindirectly with the environment.

According to another embodiment, the method may be implemented as anapplication which runs on one or more processing devices of theenvironment, and which communicates directly or indirectly with themobile device and/or camera.

According to another embodiment, the method may be implemented as anapplication which runs on one or more processing devices of a computer,such as a mobile computer or a personal computer, communicating directlyor indirectly with the mobile device and/or camera and with theenvironment.

DESCRIPTION OF THE DRAWINGS

Aspects and embodiments of the invention will now be described withrespect to the drawings, in which:

FIG. 1 shows a flowchart of an exemplary embodiment of the presentinvention,

FIG. 2 shows a flowchart of another exemplary embodiment of the presentinvention,

FIG. 3 shows an exemplary scene according to an embodiment of thepresent invention.

In the following, aspects and embodiments of the invention will beexplained with reference to the exemplary embodiments as shown in FIGS.1 to 3. The following embodiments are described with reference to usinga mobile phone, but the invention may be applied in principle with anymobile device which is associated through a specified spatialrelationship with a camera for capturing images.

DETAILED DESCRIPTION

FIG. 3 shows an exemplary scene of an environment, in which a part ofthe environment 3001 includes a front control panel 3009 and a lightsource 3007. A display device comprises a screen 3004 mounted to thefront control panel 3009. A potential visual content displayed on thescreen 3004 is a 2D map 3005, but may also be any other visual content.An environment coordinate system 3006 is associated to the environment3001.

In the exemplary scene shown in FIG. 3, the mobile device as used hereinis a mobile phone 3002 that comprises a camera 3003. Camera intrinsicparameters of the camera 3003 are provided to the mobile device.

In a potential use scenario, the environment 3001 sends environmentinformation to the mobile phone 3002 wirelessly via a computer server.The environment information includes at least part of imageryinformation (i.e. visual appearance) of the displayed 2D map 3005, aresolution of the screen 3004, a position of the screen 3004 in theenvironment coordinate system 3006, and a shape and size of the screen3004.

In another embodiment, the screen 3004 may show an iconic outline of themobile phone 3002. The iconic outline has a known position in a displaycoordinate system of the screen 3004, and thus could also have a knownposition in the environment coordinate system 3006. If the camera 3003has a known position relative to the mobile phone 3002, then the camera3003 could have a known initial position in the environment coordinatesystem 3006 when the mobile phone 3002 is placed on the positionindicated by the iconic outline on the screen 3004. This initialposition of the camera 3003 in the environment coordinate system 3006may be used for tracking the camera 3003 and/or reconstructing a part ofthe environment 3001. A position of the camera 3003 may be determinedrelative to its own initial position, i.e. determining a motion of thecamera 3003, for example based on inertial sensors attached to thecamera 3003 or based on corresponding image features between two imagescaptured by the camera 3003. Thus, the camera 3003, and accordingly themobile phone 3002 associated with the camera 3003, can be tracked in theenvironment coordinate system 3006.

The mobile device, here mobile phone 3002, has at least one or moreprocessing devices, such as one or more microprocessors and associatedcircuitry, which are commonly used in the art and not shown in theFigures, since they are internal to the mobile phone 3002. An internalprocessing device is indicated with reference number 3011 in FIG. 3.Among other tasks as commonly used and applied in the art, with regardto the present invention the processing device 3011 is configured tocommunicate with the camera 3003 and to perform tasks and steps asdescribed herein in connection with the invention, such as the steps asdescribed with reference to FIGS. 1 and 2. In this example, the mobilephone 3002 may be held by a user. Camera intrinsic parameters of thecamera 3003 are provided to the processing device 3011 or may bedetermined by a camera calibration procedure.

Moreover, the environment 3001 may comprise at least one or moreprocessing devices, such as one or more microprocessors and associatedcircuitry, which are commonly used in the art and designated in FIG. 3with reference number 3021. Among other tasks as commonly used andapplied in the art, with regard to the present invention the processingdevice 3021 is configured to display the visual content 3005 on thescreen 3004. The processing device 3021 is further applicable to performtasks and steps as described herein in connection with the invention,such as the steps as described with reference to FIGS. 1 and 2. In oneor more embodiments, the environment may be a system or collection ofsystems that include such components as a processor, circuitry, controlpanel, display screen and the like, such as a smart home system, aportable kiosk, a vehicle, a network of computer systems, and the like.

According to an embodiment, each of the processing devices 3011 and/or3021, or in any combination with each other, is appropriate and may beconfigured to perform any of the steps according to the invention asdescribed herein, such as to receive image information associated withimages captured by a camera (such as camera 3003) directly from thecamera or from another processing device, to determine a positionassociated with the mobile device in the environment coordinate systemaccording to the received image information, and to provide positioninformation indicative of the position associated with the mobile devicein the environment coordinate system. For example, such positioninformation may be provided to a control device of the environment viadirect communication, or indirectly via another processing device (suchas a server computer), to control at least one function of theenvironment according to the position.

These tasks and steps may also be performed by another one or moreprocessing devices, such as processing device 4001, which is neithercontained in the mobile phone 3002 nor in the environment 3001, but inanother device, such as a server computer 4000, communicating with themobile phone 3002 and the environment 3001, e.g. wirelessly over acomputer network. Further, it is possible that all or some of the tasksand steps according to the invention as described herein may be sharedor distributed between the processing devices 3011, 3021 and 4001.

The camera 3003 and the housing of the mobile phone 3002 have fixedpositions relative to each other. A spatial relationship between thecamera 3003 and any part of the mobile phone 3002 may be provided by themanufacturer of the mobile phone 3002 or may be calibrated as commonlyknown in the art.

Turning now to the process as described with reference to FIG. 1, in theexemplary embodiment of the invention as shown in FIG. 1, a displaydevice mounted to the environment, which is provided with an environmentcoordinate system, displays a figure on its screen in step 1001, such asa visual content in the form of a 2D map 3005 as shown in FIG. 3. Theenvironment sends environment information from the environment to themobile phone that comprises a camera via Internet in step 1002. Thecamera of the mobile phone captures an image of the figure displayed onthe display device in step 1003. In the exemplary scene shown in FIG. 3,the camera 3003 captures an image of the displayed 2D map 3005. Theimage may include only a part of the displayed 2D map 3005.

Step 1004 determines, in this embodiment on the mobile phone, a cameraposition in the environment coordinate system while the camera capturesthe image. In the exemplary scene shown in FIG. 3, a camera position ofthe camera 3003 in the environment coordinate system 3006 while thecamera 3003 captures the image can be determined, which is performed onthe processing device 3011 of mobile phone 3002 according to receivedimage information associated with the captured image. The determinationof the camera position could be based on matching the image captured bythe camera 3003 and the imagery information of the 2D map 3005 receivedat the mobile phone 3002. From the determined camera position, theposition associated with the mobile device 3002 in the environmentcoordinate system 3006 may be determined which may be used as an inputfor controlling at least one function of the environment. A knownspatial relationship between the camera 3003 and any part of the mobiledevice 3002 may be used for this determination.

Step 1005 sends the determined camera position from the mobile phone tothe environment. In the exemplary scene shown in FIG. 3, the cameraposition of the camera 3003 in the environment coordinate system 3006 isprovided from the mobile phone 3002 to the environment 3001.

The environment then may control at least one function of theenvironment according to the received camera position in step 1006. Inthe exemplary scene shown in FIG. 3, the processing device 3021 ofenvironment 3001 may control an illuminating direction of the lightsource 3007. The light source 3007 could be directed to illuminate thearea 3008 along a direction defined by the camera position of the camera3003.

FIG. 2 shows a flowchart of another exemplary embodiment. Step 2001displays a figure on a display device mounted to an environment providedwith an environment coordinate system. The environment sends environmentinformation to a mobile phone that comprises a camera in step 2002. Thecamera of the mobile phone captures a first image of the displayedfigure in step 2003. Step 2004 determines, on the mobile phone, a cameraposition P1 in the environment coordinate system while the cameracaptures the first image according to the first image and the displayedfigure. In step 2005, the camera, i.e. the mobile phone, is moved toanother position that is different from the position P1. The camera ofthe mobile phone captures a second image of the displayed figure in step2006. Step 2007 determines, on the mobile phone, a camera position P2 inthe environment coordinate system while the camera captures the secondimage according to image information of the second image and thedisplayed figure. Step 2008 reconstructs a model of at least part of theenvironment on the mobile phone based on the image information of thefirst image and the second image and the camera positions P1 and P2.This could be realized using a triangulation method.

In step 2009, the camera is moved. The camera captures a third image instep 2010. Step 2011 determines, on the mobile phone, a camera positionP3 in the environment coordinate system while the camera captures thethird image according to the model. This could be implemented based on aSLAM method. In step 2012, it is determined if the position P3 is adesired position. The desired position indicates that a user wants toconfigure or control a function of the environment according to thedesired position. For example, the user may read a book. The desiredposition may indicate a center position of an area around the book wherean ambient light system of the environment should illuminate. If it isnot a desired position, then go to step 2008, and extend the modelaccording to the third image using, e.g., the SLAM method. On the otherhand, if it is the desired position, then send the position P3 from themobile phone to the environment (step 2013). The environment is thencapable to control at least one function of the environment according tothe received position P3 (step 2014).

A method for providing position information for controlling at least onefunction of a vehicle provided with a vehicle coordinate system mayinclude receiving image information associated with at least one imageof at least one part of the vehicle captured by a camera of a mobiledevice, the mobile device being separate from the vehicle, determining aposition associated with the mobile device in the vehicle coordinatesystem according to the image information associated with the at leastone image, and providing position information indicative of the positionassociated with the mobile device in the vehicle coordinate system tocontrol at least one function of the vehicle according to the position.The method may also include providing or receiving a position of the atleast one part of the vehicle in the vehicle coordinate system. The atleast one part of the vehicle comprises a visual content displayed on adisplay device mounted to the vehicle. Determining the positionassociated with the mobile device in the vehicle coordinate system mayinclude determining a camera position of the camera relative to the atleast one part of the vehicle when capturing the at least one imageaccording to the image information associated with the at least oneimage, and determining the position associated with the mobile device inthe vehicle coordinate system according to the camera position and theposition of the at least one part of the vehicle in the vehiclecoordinate system. The at least one image is a first image and thecamera may be a first camera, and determining the position associatedwith the mobile device in the vehicle coordinate system may includereceiving image information associated with a second image of the atleast one part of the vehicle captured by a second camera of the mobiledevice, and reconstructing a model of the at least one part of thevehicle according to the image information associated with the first andsecond images.

The method may also include receiving image information associated witha third image captured by a third camera of the mobile device, whereinthe third image captures at least a part of the vehicle represented byat least part of the reconstructed model, determining a camera positionof the third camera relative to the reconstructed model when capturingthe third image according to the at least part of the reconstructedmodel and image information associated with the third image, anddetermining the position associated with the mobile device in thevehicle coordinate system according to the camera position of the thirdcamera. The method may also include determining a camera position of thefirst camera relative to the at least one part of the vehicle whencapturing the first image according to at least part of the imageinformation associated with the first image, and determining a cameraposition of the second camera relative to the at least one part of thevehicle when capturing the second image according to at least part ofthe image information associated with the second image. The at least onepart of the vehicle may be a first part of the vehicle and the methodmay also include receiving image information associated with a fourthimage captured by a fourth camera of the mobile device, wherein thefourth image captures at least part of the first part of the vehicle andat least a second part of the vehicle, and extending the reconstructedmodel to include the first part of the vehicle and the second part ofthe vehicle. The reconstructed model may at least describes depthinformation of at least part of the vehicle. At least two of the firstcamera, the second camera, the third camera, and the fourth camera maybe the same camera. At least two of the first camera, the second camera,the third camera, and the fourth camera may be different cameras. The atleast one function may be a function which is related to at least one ofa sound system of the vehicle, an ambient light system of the vehicle,and an air conditioning system of the vehicle, and which is controlledaccording to the position.

A method for providing movement information for controlling at least onefunction of a vehicle provided with a vehicle coordinate system mayinclude receiving image information associated with at least two imagescaptured by at least one camera of a mobile device, the mobile devicebeing separate from the vehicle, each of the at least two imagescapturing at least one part of the vehicle, determining a movementassociated with the mobile device relative to the vehicle coordinatesystem according to the image information associated with the at leasttwo images, and providing movement information indicative of themovement associated with the mobile device relative to the vehiclecoordinate system to control at least one function of the vehicleaccording to the movement. The movement associated with the mobiledevice relative to the vehicle coordinate system may be determinedaccording to at least two positions associated with the mobile device inthe vehicle coordinate system. The movement associated with the mobiledevice relative to the vehicle coordinate system may be determinedaccording to at least two camera positions of the at least one camerawhile capturing the at least two images. The movement associated withthe mobile device relative to the vehicle coordinate system may bedetermined according to image positions of the at least one part of thevehicle captured in the each of the at least two images. The at leastone function may be a function controlled based on a determined movementassociated with the mobile device, particularly a function which isrelated to at least one of a sound system, a lighting system and an airconditioning of the vehicle controlled according to the movement of themobile device.

A system for providing position information for controlling at least onefunction of a vehicle provided with a vehicle coordinate system mayinclude a first processing device configured to receive imageinformation associated with at least one image of at least one part ofthe vehicle captured by a camera of a mobile device, the mobile devicebeing separate from the vehicle, the first processing device configuredto determine a position associated with the mobile device in the vehiclecoordinate system according to the image information associated with theat least one image, and a second processing device configured to provideposition information indicative of the position associated with themobile device in the vehicle coordinate system to control at least onefunction of the vehicle according to the position.

A system for providing movement information for controlling at least onefunction of a vehicle provided with a vehicle coordinate system, mayinclude a first processing device configured to receive imageinformation associated with at least two images captured by at least onecamera of a mobile device, the mobile device being separate from thevehicle, and each of the at least two images capturing at least one partof the vehicle, the first processing device configured to determine amovement associated with the mobile device relative to the vehiclecoordinate system according to the image information associated with theat least two images, and a second processing device configured toprovide movement information indicative of the movement associated withthe mobile device relative to the vehicle coordinate system to controlat least one function of the vehicle according to the movement. Thefirst processing device and the second processing device may be the sameprocessing device. At least one of the first and second processingdevices may be comprised in the mobile device. At least one of the firstand second processing devices may be comprised in the vehicle. At leastone of the first and second processing devices may be comprised in acomputer device which communicates with the mobile device and thevehicle. The mobile device may be a hand held device, particularly amobile phone, a tablet computer or a mobile computer.

A computer program product may include software code sections which areadapted to perform a method as described above when loaded into theinternal memory of a processing device.

The steps and system components as described above related to the firstaspect of a method and system for providing position information forcontrolling at least one function of an environment may analogously beapplied with same or similar components in connection with the secondaspect of a method and system for providing movement information forcontrolling at least one function of an environment, with themodification of providing image information associated with at least twoimages captured by at least one camera of a mobile device, and thefollowing steps considering this image information as described above.Further, the components, such as mobile device, camera, and/orprocessing devices, as described above may also be configured to performthe method of the second aspect for providing movement information forcontrolling at least one function of an environment.

Throughout this document it is described that image informationassociated with an image is provided or received. It is known to theskilled person that this may include providing or receiving anyprocessed or non-processed information (version) of an image, part of animage and/or features of an image which allows for position or poseestimation. The invention does not require providing or receiving anyraw image data. Processing thereby includes any one of compression (e.g.JPEG, PNG, ZIP), encryption (e.g. RSA encryption, Schnorr signature,El-Gamal encryption, PGP), conversion to another color space orgrayscale, cropping or scaling the image or conversion into a sparserepresentation based on feature descriptors, extraction, and theircombinations. All these image processing methods can optionally beperformed and are covered by the terminology of image information of orassociated with an image.

1. A non-transitory computer readable medium comprising computerreadable code executable by one or more processors to: receive an imagecaptured by a camera of a mobile device in a vehicle, wherein the mobiledevice is separate from the vehicle, and wherein the image is receivedas part of a request from the mobile device to for control of afunctionality of the vehicle; determine a location of the mobile devicewithin the vehicle based on the image; and control direction of thefunctionality of the vehicle for the determined location of the mobiledevice within the vehicle.
 2. The non-transitory computer readablemedium of claim 1, wherein location of the mobile device is determinedin a vehicle coordinate system for the vehicle.
 3. The non-transitorycomputer readable medium of claim 1, wherein the image comprises aportion of the vehicle, and wherein the computer readable code todetermine the location of the mobile device within the vehicle furthercomprises computer readable code to: determine a camera position of thecamera relative to the portion of the vehicle according to the image,and determine the location of the mobile device based on the determinedcamera position relative to the portion of the vehicle.
 4. Thenon-transitory computer readable medium of claim 3, wherein the computerreadable code to determine a camera position of the camera relative tothe portion of the vehicle comprises: obtaining depth information forthe portion of the vehicle from a depth sensor of the mobile device. 5.The non-transitory computer readable medium of claim 3, wherein thecomputer readable code to determine a camera position of the camerarelative to the portion of the vehicle comprises: obtaining depthinformation for the portion of the vehicle from a model of the vehicle.6. The non-transitory computer readable medium of claim 3, wherein theportion of the vehicle comprises a display device of the vehicle.
 7. Thenon-transitory computer readable medium of claim 6, wherein the computerreadable code to determine a camera position of the camera relative tothe portion of the vehicle comprises: identifying, in the image, avisual content displayed on the display device, wherein the location ofthe mobile device is further determined based on the visual content asit is presented in the image.
 8. A system comprising: one or moreprocessors; and a computer readable medium comprising computer readablecode executable by the one or more processors to: receive an imagecaptured by a camera of a mobile device in the vehicle, wherein themobile device is separate from the vehicle, and wherein the image isreceived as part of a request from the mobile device to for control of afunctionality of the vehicle; determine a location of the mobile devicewithin the vehicle based on the image; and control direction of thefunctionality of the vehicle for the determined location of the mobiledevice within the vehicle.
 9. The system of claim 8, wherein location ofthe mobile device is determined in a vehicle coordinate system for thevehicle.
 10. The system of claim 8, wherein the image comprises aportion of the vehicle, and wherein the computer readable code todetermine the location of the mobile device within the vehicle furthercomprises computer readable code to: determine a camera position of thecamera relative to the portion of the vehicle according to the image,and determine the location of the mobile device based on the determinedcamera position relative to the portion of the vehicle.
 11. The systemof claim 10, wherein the computer readable code to determine a cameraposition of the camera relative to the portion of the vehicle comprises:obtaining depth information for the portion of the vehicle from a depthsensor of the mobile device.
 12. The system of claim 10, wherein thecomputer readable code to determine a camera position of the camerarelative to the portion of the vehicle comprises: obtaining depthinformation for the portion of the vehicle from a model of the vehicle.13. The system of claim 10, wherein the portion of the vehicle comprisesa display device of the vehicle.
 14. The system of claim 13, wherein thecomputer readable code to determine a camera position of the camerarelative to the portion of the vehicle comprises: identifying, in theimage, a visual content displayed on the display device, wherein thelocation of the mobile device is further determined based on the visualcontent as it is presented in the image.
 15. A method comprising:receiving an image captured by a camera of a mobile device in a vehicle,wherein the mobile device is separate from the vehicle, and wherein theimage is received as part of a request from the mobile device to forcontrol of a functionality of the vehicle; determining a location of themobile device within the vehicle based on the image; and controllingdirection of the functionality of the vehicle for the determinedlocation of the mobile device within the vehicle.
 16. The method ofclaim 15, wherein the image comprises a portion of the vehicle, andwherein the method further comprises: determining a camera position ofthe camera relative to the portion of the vehicle according to theimage, and determining the location of the mobile device based on thedetermined camera position relative to the portion of the vehicle. 17.The method of claim 16, further comprising: obtaining depth informationfor the portion of the vehicle from a depth sensor of the mobile device.18. The method of claim 16, wherein determining a camera position of thecamera relative to the portion of the vehicle comprises: obtaining depthinformation for the portion of the vehicle from a model of the vehicle.19. The method of claim 16, wherein the portion of the vehicle comprisesa display device of the vehicle.
 20. The method of claim 19, whereindetermining a camera position of the camera relative to the portion ofthe vehicle comprises: identifying, in the image, a visual contentdisplayed on the display device, wherein the location of the mobiledevice is further determined based on the visual content as it ispresented in the image.